Comments for MEDB 5501, Week 8

Review multiple comparisons issue

Type I error: rejecting the null hypothesis when the null hypothesis is true.
- Multiple simultaneous hypotheses increase the Type I error rate.

Bonferroni inequality for two simultaneous hypotheses

Define

\(E_1\) = Type I error for Hypothesis 1
\(E_2\) = Type I error for Hypothesis 2
- \(P[E_1\ \cup\ E_2]=P[E_1]+P[E_2]-P[E_1\ \cap\ E_2]\)
- \(P[E_1\ \cup\ E_2]\ \le\ P[E_1]+P[E_2]\)
- \(P[E_1\ \cup\ E_2]\ \le\ P[E_1]+P[E_2]\)
- \(P[E_1\ \cup\ E_2]\ \le\ 2 \alpha\)

Bonferroni adjustment

For m hypotheses
- \(P[E_1\ \cup\ ...\ \cup E_m]\ \le\ m \alpha\)
Test each hypothesis at \(\alpha/m\)
- Preserves overall Type I error rate
Example, 3 simultaneous hypotheses
- Reject H0 if p-value < 0.0133

Controversies over Bonferroni adjustment

Increases Type II errors
Impractical for large values of m
Works poorly for highly correlated tests
Ambiguity in definition of “simultaneous” hypotheses

Alternatives to Bonferroni adjustment

False discovery rate
Designation of primary and secondary outcomes
Subjective assessment of simultaneous hypotheses

Review two-sample t-test

\(H_0:\ \mu_1=\mu_2\)
\(H_1:\ \mu_1 \ne \mu_2\)
- Or a one tailed alternative
\(T=\frac{\bar X_1-\bar X_2}{SE(\bar X_1-\bar X_2)}\)
- Accept H0 if T is close to zero.

What to do with three or more groups?

\(H_0:\ \mu_1=\mu_2=...=\mu_k\)
\(H_1:\ \mu_i \ne \mu_j\) for some i, j
- Note: one-tailed test is tricky.
Accept H0 if the F ratio (defined below) is close to 1.

Important assumptions

Same as independent-samples t-test
- Normality
- Equal variances
- Independence

How to check assumptions

Boxplots
Analysis of residuals, \(e_{ij}\)
- \(e_{ij}\)= Observed - Predicted
- \(e_{ij}= Y_{ij}-\bar{Y}_i\)

Tukey post hoc tests

If you reject H0, which values are unequal
- With k groups, there are k(k-1)/2 comparisons
Studentized range (Tukey test)
- Requires equal sample sizes per group
- Uses harmonic mean approximation for unequal sample sizes.
  - Do not use harmonic means if seriously different sample sizes.

Alternatives to Tukey post hoc tests

Bonferroni adjustment
- Works for unequal sample sizes per group
- Works for unequal variances
Dunnett’s test
- Treatment versus multiple controls
Scheffe’s test
- Works for complex comparison
  - Example \(\mu_1\ vs.\ \frac{\mu_2+\mu_3+\mu_4}{3}\)

Artificial data

Scatterplot

Total SS = 452

Within SS = 116

Between SS = 336

Degrees of freedom

For Total SS, df = 10
For Within SS, df = 8
For Between SS, df = 2
In general,
- N = number of observations
- k = number of groups
  - Total df = N-1
  - Within df = N-k
  - Between df = k-1

ANOVA table, 1 of 4

ANOVA table, 2 of 4

ANOVA table, 3 of 4

ANOVA table, 4 of 4

ANOVA table from the general linear model

Regression approach to ANOVA, 1 of 4

Regression approach to ANOVA, 2 of 4

Regression approach to ANOVA, 3 of 4

Regression approach to ANOVA, 4 of 4

Predicted values and residuals, 1 of 2

Predicted values and residuals, 2 of 2

What belongs in an interpretation

“Sixty-six women with osteoporosis were alternately assigned to one of three treatment groups: Group 1 (n=22), group 2 (n=22), and controls (n=22). After 6 weeks, the change in bone mineral density from baseline was measured. Analysis with one-way ANOVA indicated a statistically significant difference between the groups (F_2,63=61.07, P < 0.001). Further analysis with Tukey’s pair-wise comparison procedure to control for multiple testing revealed that the mean change (+/- SD) of group 2 (1.6 g/cm^2 +/-0.2) was significantly greater than that of group 1 (1.1 g/cm^2 +/- 0.2) and that of the controls (1.0 g/cm^2 +/- 0.2) with an overall alpha level of 0.05.”

Lang and Secic, How to Report Statistics in Medicine, p110.

Identify the dependent variable

“Sixty-six women with osteoporosis were alternately assigned to one of three treatment groups: Group 1 (n=22), group 2 (n=22), and controls (n=22). After 6 weeks, the change in bone mineral density from baseline was measured. Analysis with one-way ANOVA indicated a statistically significant difference between the groups (F_2,63=61.07, P < 0.001). Further analysis with Tukey’s pair-wise comparison procedure to control for multiple testing revealed that the mean change (+/- SD) of group 2 (1.6 g/cm^2 +/-0.2) was significantly greater than that of group 1 (1.1 g/cm^2 +/- 0.2) and that of the controls (1.0 g/cm^2 +/- 0.2) with an overall alpha level of 0.05.”

Lang and Secic, How to Report Statistics in Medicine, p110.

Summarize each group

“Sixty-six women with osteoporosis were alternately assigned to one of three treatment groups: Group 1 (n=22), group 2 (n=22), and controls (n=22). After 6 weeks, the change in bone mineral density from baseline was measured. Analysis with one-way ANOVA indicated a statistically significant difference between the groups (F_2,63=61.07, P < 0.001). Further analysis with Tukey’s pair-wise comparison procedure to control for multiple testing revealed that the mean change (+/- SD) of group 2 (1.6 g/cm^2 +/-0.2) was significantly greater than that of group 1 (1.1 g/cm^2 +/- 0.2) and that of the controls (1.0 g/cm^2 +/- 0.2) with an overall alpha level of 0.05.”

Lang and Secic, How to Report Statistics in Medicine, p110.

Explain how subjects were assigned

“Sixty-six women with osteoporosis were alternately assigned to one of three treatment groups: Group 1 (n=22), group 2 (n=22), and controls (n=22). After 6 weeks, the change in bone mineral density from baseline was measured. Analysis with one-way ANOVA indicated a statistically significant difference between the groups (F_2,63=61.07, P < 0.001). Further analysis with Tukey’s pair-wise comparison procedure to control for multiple testing revealed that the mean change (+/- SD) of group 2 (1.6 g/cm^2 +/-0.2) was significantly greater than that of group 1 (1.1 g/cm^2 +/- 0.2) and that of the controls (1.0 g/cm^2 +/- 0.2) with an overall alpha level of 0.05.”

Lang and Secic, How to Report Statistics in Medicine, p110.

Describe the statistical test

“Sixty-six women with osteoporosis were alternately assigned to one of three treatment groups: Group 1 (n=22), group 2 (n=22), and controls (n=22). After 6 weeks, the change in bone mineral density from baseline was measured. Analysis with one-way ANOVA indicated a statistically significant difference between the groups (F_2,63=61.07, P < 0.001). Further analysis with Tukey’s pair-wise comparison procedure to control for multiple testing revealed that the mean change (+/- SD) of group 2 (1.6 g/cm^2 +/-0.2) was significantly greater than that of group 1 (1.1 g/cm^2 +/- 0.2) and that of the controls (1.0 g/cm^2 +/- 0.2) with an overall alpha level of 0.05.”

Lang and Secic, How to Report Statistics in Medicine, p110.

Specify the alpha level

“Sixty-six women with osteoporosis were alternately assigned to one of three treatment groups: Group 1 (n=22), group 2 (n=22), and controls (n=22). After 6 weeks, the change in bone mineral density from baseline was measured. Analysis with one-way ANOVA indicated a statistically significant difference between the groups (F_2,63=61.07, P < 0.001). Further analysis with Tukey’s pair-wise comparison procedure to control for multiple testing revealed that the mean change (+/- SD) of group 2 (1.6 g/cm^2 +/-0.2) was significantly greater than that of group 1 (1.1 g/cm^2 +/- 0.2) and that of the controls (1.0 g/cm^2 +/- 0.2) with an overall alpha level of 0.05.”

Lang and Secic, How to Report Statistics in Medicine, p110.

Specify the F statistic with degrees of freedom

“Sixty-six women with osteoporosis were alternately assigned to one of three treatment groups: Group 1 (n=22), group 2 (n=22), and controls (n=22). After 6 weeks, the change in bone mineral density from baseline was measured. Analysis with one-way ANOVA indicated a statistically significant difference between the groups (F_2,63=61.07, P < 0.001). Further analysis with Tukey’s pair-wise comparison procedure to control for multiple testing revealed that the mean change (+/- SD) of group 2 (1.6 g/cm^2 +/-0.2) was significantly greater than that of group 1 (1.1 g/cm^2 +/- 0.2) and that of the controls (1.0 g/cm^2 +/- 0.2) with an overall alpha level of 0.05.”

Lang and Secic, How to Report Statistics in Medicine, p110.

State the p-value

“Sixty-six women with osteoporosis were alternately assigned to one of three treatment groups: Group 1 (n=22), group 2 (n=22), and controls (n=22). After 6 weeks, the change in bone mineral density from baseline was measured. Analysis with one-way ANOVA indicated a statistically significant difference between the groups (F_2,63=61.07, P < 0.001). Further analysis with Tukey’s pair-wise comparison procedure to control for multiple testing revealed that the mean change (+/- SD) of group 2 (1.6 g/cm^2 +/-0.2) was significantly greater than that of group 1 (1.1 g/cm^2 +/- 0.2) and that of the controls (1.0 g/cm^2 +/- 0.2) with an overall alpha level of 0.05.”

Lang and Secic, How to Report Statistics in Medicine, p110.

Describe any post hoc analyses

“Sixty-six women with osteoporosis were alternately assigned to one of three treatment groups: Group 1 (n=22), group 2 (n=22), and controls (n=22). After 6 weeks, the change in bone mineral density from baseline was measured. Analysis with one-way ANOVA indicated a statistically significant difference between the groups (F_2,63=61.07, P < 0.001). Further analysis with Tukey’s pair-wise comparison procedure to control for multiple testing revealed that the mean change (+/- SD) of group 2 (1.6 g/cm^2 +/-0.2) was significantly greater than that of group 1 (1.1 g/cm^2 +/- 0.2) and that of the controls (1.0 g/cm^2 +/- 0.2) with an overall alpha level of 0.05.”

Lang and Secic, How to Report Statistics in Medicine, p110.

Yeast activation experiment

“To shorten the time it takes him tomake his favorite pizza, a student designed an experiment to test the effect of sugar and milk on the activation times for baking yeast. Specifically, he tested four different recipes and measured how many seconds it took for the same amount of dough to rise to the top of a bowl. He randomized the order of the recipes and replicated each treatment 4 times.”

DASL, Activating baking yeast

Yeast analysis, descriptive statistics

Yeast analysis, ANOVA table

Yeast analysis, parameter estimates

Yeast analysis, Tukey post hoc, 1 of 2

Yeast analysis, Tukey post hoc, 2 of 2

Yeast analysis, interpretation, 1 of 3

The average activation times were compared using oneway ANOVA with a Tukey post hoc follow-up test.
All tests used a two-sided alpha level of 0.05.

Yeast analysis, interpretation, 2 of 3

There were statistically significant differences (p < 0.001) in the activation times of the four recipes. Recipes D and B had similar average activation times which were significantly faster than Recipe A, which itself was significantly faster than recipe C.

Yeast analysis, interpretation, 3 of 3

Recipes D and B were similar and had the fastest activation times. These differences were more than four minutes faster than the next best recipe which represents a practical as well as statistically significant result.

Comments for MEDB 5501, Week 8

Review multiple comparisons issue

Bonferroni inequality for two simultaneous hypotheses

Bonferroni adjustment

Controversies over Bonferroni adjustment

Alternatives to Bonferroni adjustment

Review two-sample t-test

What to do with three or more groups?

Important assumptions

How to check assumptions

Tukey post hoc tests

Alternatives to Tukey post hoc tests

Artificial data

Scatterplot

Total SS = 452

Within SS = 116

Between SS = 336

Degrees of freedom

ANOVA table, 1 of 4

ANOVA table, 2 of 4

ANOVA table, 3 of 4

ANOVA table, 4 of 4

ANOVA table from the general linear model

Regression approach to ANOVA, 1 of 4

Regression approach to ANOVA, 2 of 4

Regression approach to ANOVA, 3 of 4

Regression approach to ANOVA, 4 of 4

Predicted values and residuals, 1 of 2

Predicted values and residuals, 2 of 2

What belongs in an interpretation

Identify the dependent variable

Summarize each group

Explain how subjects were assigned

Describe the statistical test

Specify the alpha level

Specify the F statistic with degrees of freedom

State the p-value

Describe any post hoc analyses

Yeast activation experiment

Yeast analysis, descriptive statistics

Yeast analysis, ANOVA table

Yeast analysis, parameter estimates

Yeast analysis, Tukey post hoc, 1 of 2

Yeast analysis, Tukey post hoc, 2 of 2

Yeast analysis, interpretation, 1 of 3

Yeast analysis, interpretation, 2 of 3

Yeast analysis, interpretation, 3 of 3

Recode into different variables dialog box

Old and new Values dialog box

General Linear Model | Univariate dialog box

Descriptives output

Boxplot

Analysis of variance table

Analysis of variance table with irrelevant rows removed

Parameter estimates

Tukey post hoc test, 1 of 7

Tukey post hoc test, 2 of 7

Tukey post hoc test, 3 of 7

Tukey post hoc test, 4 of 7

Tukey post hoc test, 5 of 7

Tukey post hoc test, 6 of 7

Tukey post hoc test, 7 of 7

Checking assumptions, boxplots

Checking assumptions, descriptive statistics

Residual analysis, 1 of 2

Residual analysis, 2 of 2

Analysis of school year, 1 of 4

Analysis of school year, 2 of 4

Analysis of school year, 3 of 4

Analysis of school year, 4 of 4